Problem

In a 3-DC topology — primary in dc1, standby in dc2, monitor in dc3 — cutting the dc1↔dc2 link while both nodes remain reachable from the monitor leaves synchronous_standby_names set on the primary. Every COMMIT then hangs indefinitely waiting for an acknowledgement from a standby that can no longer be reached.

Closes #997.

Root cause

The FSM had no rule to transition the primary out of the primary state when synchronous replication was stalled. The monitor could see that the standby was unreachable from the primary (via pg_stat_replication reporting sync_state = 'unknown') but took no action.

Fix

Monitor side (src/monitor/):

• Add a replication_stall_since column to pgautofailover.node. The monitor sets it to now() the first time a primary reports sync_state = unknown (replication partner unreachable) and clears it when replication recovers.
• Add a region column to pgautofailover.node (populated from --region on pg_autoctl create postgres). This lets operators label nodes with their DC location for observability.
• Add a GUC pgautofailover.replication_stall_timeout (default 10 s). When now() - replication_stall_since > timeout, the FSM assigns wait_primary, which clears synchronous_standby_names and unblocks writes.
• Extension bumped to 2.3 with an upgrade path from 2.2.

Client side (src/bin/pg_autoctl/):

• --region / -G flag added to pg_autoctl create postgres and wired through keeper_config, monitor_register_node, and nodespec.
• pg_autoctl watch updated to display the region column.

Test (tests/tap/specs/replication_stall_3dc.pgaf):

New end-to-end regression test using the pgaftest framework. The test models the 3-DC scenario in a single Docker Compose stack: the monitor is always reachable from both nodes, but docker network disconnect severs the dc1↔dc2 replication link. Six steps verify:

1. Sync replication is active (synchronous_standby_names = 'ANY 1 (…)')
2. The dc1↔dc2 link is cut
3. The monitor assigns wait_primary within ~20 s (well within the 60 s timeout)
4. synchronous_standby_names is cleared; writes no longer hang
5. Link restored; standby re-joins
6. Row written during the stall has replicated

Verified locally, 6/6 steps passing against PG17.

Watch the fix work, interactively

Bring the exact topology up and drive it step by step (pulls/builds images on first run, ~30s):

pgaftest tmux tests/tap/specs/replication_stall_3dc.pgaf

This opens a 3-pane tmux session: docker compose logs -f (top), pg_autoctl watch — the live FSM state table (middle), an interactive shell (bottom). From the shell:

# See the six steps and where you are
pgaftest show steps

# Preview the DSL commands the next step will run before running it
pgaftest show step

# Confirm sync replication is active: synchronous_standby_names = 'ANY 1 (...)'
pgaftest step

# Sever the dc1<->dc2 link -- watch the middle pane: node1 stays "primary"
# reporting an empty sync_state
pgaftest step
pgaftest show state

# Within ~10-20s (replication_stall_timeout), node1 flips to wait_primary
# in the live watch pane -- this step just asserts what you can already see
pgaftest step

# Confirm writes actually unblocked: synchronous_standby_names is now empty
pgaftest step
pgaftest sql node1 { SHOW synchronous_standby_names; }

# Restore the link; node2 rejoins as secondary, node1 confirms primary
pgaftest step

# Confirm replication caught back up, including the row written during the stall
pgaftest step

# Tear down
pgaftest cluster down

Each pgaftest step (no argument) auto-advances to the next pending step in the spec's sequence; pgaftest show steps at any point lists all six with * marking which one runs next.

To see the bug rather than the fix: the spec file itself doesn't exist before this PR, but the FSM rule it exercises is a single, self-contained if block in ProceedGroupStateFromContext() (src/monitor/group_state_machine.c). Comment that block out, rebuild the monitor image, and rerun the spec — step 3 times out instead of passing (node1 stays at primary with synchronous_standby_names still set), and step 4's write hangs forever (interrupt it and pgaftest cluster down to clean up).

Reusing this for future bug reports

The spec above didn't start as a regression test — it started as the smallest .pgaf file that reliably reproduced the reported bug. That's a generally useful pattern for triaging issues in this repo, not just for this one:

1. When opening an issue, attach a minimal .pgaf file describing just the topology and the manual steps that trigger the problem (a cluster{} block, a setup{} that gets to the interesting state, and network/sql/exec commands reproducing the failure — no step{}/sequence needed yet). pgaftest show compose <spec.pgaf> lets the reporter sanity-check the generated Docker Compose file without needing to actually understand pgaftest's internals.
2. When triaging, a maintainer runs pgaftest tmux <attached-spec.pgaf> and gets the exact reported topology live in under a minute, with pg_autoctl watch showing FSM state in real time and docker compose logs -f showing every node's logs — no need to describe the bug in prose or guess at timing from a text report, and no need to reconstruct the environment from scratch.
3. While diagnosing, pgaftest sql <node> { ... }, pgaftest network disconnect/connect <node>, and pgaftest show state let the maintainer poke at the live cluster interactively — inject the same fault differently, check intermediate state, confirm a hypothesis — without editing the spec or restarting the stack.
4. Once the root cause is understood and fixed, the reporter's manual steps get turned into named step{}s with wait until/assert/expect assertions and a sequence (exactly what happened between the original issue-2 nodes + witness = 3 data centers (problem case detected) #997 repro and replication_stall_3dc.pgaf above), and pgaftest run <spec.pgaf> gives the same scenario a permanent, deterministic, TAP-emitting place in CI — so the exact bug that was reported can never silently come back.

The whole loop — reporter's repro spec straight through to CI regression test — uses the same file and the same tool the whole way, which is the main advantage over a prose bug report: nobody has to translate "here's what I did" into "here's a test" by hand.

Rebase notes

Rebased onto current main (post-#1150) on 2026-07-22; the earlier squashed pgaftest-development commits this branch used to carry (CLI restructure, pg_autoctl watch column-count fix, installcheck expected-output fixes, etc.) are already on main via #1150 and are no longer part of this PR's diff. Two pre-existing bugs were fixed while rebasing: a stray trailing backslash in the Dockerfile that broke every Docker build, and an incomplete 2.2→2.3 version bump (PG_AUTOCTL_EXTENSION_VERSION in defaults.h was left at "2.2", which made every pg_autoctl start try to downgrade the extension). tests/tap/specs/upgrade.pgaf's hardcoded version strings were also updated to match (its PREV_TAG auto-detection now resolves to v2.2 instead of v2.1, since it always targets the newest released tag below the current in-development version).